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Abstract 


Performance  Based  Logistics  is  an  acquisition  reform  that  is  intended  to  improve 
weapon  systems  logistics  by  reducing  cost,  improving  reliability,  and  reducing  footprint. 
It  is  an  extension  of  a  broad  process  of  rationalizing,  and  in  many  cases  outsourcing 
government  services.  As  with  other  examples  of  governmental  service  outsourcing 
measurement  issues  arise,  in  the  gap  between  governmental  objectives  and  service 
measurement,  and  in  the  contrast  between  clear  profit-centered  vendor  metrics,  and 
more  complex  mission-oriented  governmental  metrics.  Beyond  this  however,  PBL 
presents  new  challenges  to  the  relationship  between  governmental  agencies  and  their 
service  vendors.  In  many  cases,  weapons  systems  logistical  support  involves  levels  of 
operational  risk  that  are  more  difficult  to  measure  and  more  difficult  to  value  than  other 
government  services.  We  discuss  the  implications  of  operational  risk  and  other 
measurement  issues  on  PBL  implementation. 

Keywords:  Performance  Based  Logistics,  PBL,  Acquisition  Reform,  Weapon 
Systems  Logistics,  Outsourcing,  Risk  Management,  Future  Logistics  Enterprise 
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Introduction 


Performance  Based  Service  Acquisition  is  a  Department  of  Defense  (DoD) 
acquisition  reform  that  has  had  noted  success  in  reducing  cost  and  streamlining  the 
management  of  non-core  government  service  capabilities  (Office  of  the  Deputy  Under 
Secretary  of  Defense  -  Defense  Acquisition  Reform  [OSD-DAR],  2000).  The  guiding 
principle  in  Performance  Based  Service  Acquisition  is  that  when  an  outside  vendor 
exists  who  can  perform  a  service  more  effectively  than  a  government  user  could 
organically,  that  user  should  specify  measurable  outcomes  to  a  service  vendor,  and 
allow  the  vendor  to  best  determine  the  appropriate  processes  (how)  of  delivering  the 
service.  In  adopting  this  reform,  DoD  has  been  influenced  by  the  perceived  success  of 
outsourcing  in  the  private  sector.  Firms  have  witnessed  gains  from  devolving  non-core 
activities  to  suppliers,  while  achieving  high  levels  of  transparency  so  that  visibility  of 
inventory  and  information  is  maintained  throughout  the  supply  chain  (Spekman  &  Davis, 
2004). 


Performance  Based  Logistics  (PBL)  is  an  extension  of  Performance  Based 
Service  Acquisition  aimed  at  the  logistic  services  for  major  weapon  systems.  The 
Quadrennial  Defense  Review  mandated  DoD  to  implement  PBL,  in  order  to  “compress 
the  supply  chain  and  improve  readiness  for  major  weapons  systems  and  commodities.” 
(Office  of  the  Secretary  of  Defense,  2001 :  56)  PBL  is  intended  to  reduce  lifecycle  cost, 
increase  readiness,  improve  reliability  and  reduce  the  logistical  footprint  of  weapon 
systems.  A  number  of  case  studies  of  successful  PBL  initiatives  are  available  (e.g., 
Candreva,  et  al.,  2001 ). 

This  paper  takes  for  granted  the  ubiquitous  nature  of  PBL  initiatives,  and  takes 
as  its  starting  point  the  question  of  how  best  to  measure  the  degree  of  their  success.  In 
support  of  our  prescriptions  for  measurement,  we  will  draw  not  only  on  successful  best 
practice,  but  also  on  the  underlying  logic  and  justification  of  outsourcing,  as  laid  out  in 
the  economics  and  management  literature.  While  PBL  prescriptions  from  OSD  are 
always  careful  to  explain  that  a  PBL  initiative  may  result  in  the  selection  of  an  ‘organic 
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contractor’  (i.e.,  another  DoD  command),  actual  instances  of  ‘organic  contractors’  are 
fairly  rare,  and  in  any  event,  some  of  the  same  measurement  issues  arise  regardless  of 
the  blend  of  private  sector  and  organic  resources. 

While  measuring  the  performance  of  ongoing  PBL  initiatives  is  our  starting  point, 
we  also  intend  this  paper  to  inform  valuation  questions.  From  the  initial  question  of 
whether  to  bring  forward  a  weapon  system  or  a  major  component  of  a  weapon  system 
as  a  candidate  for  PBL,  to  later  design  questions  of  ‘what  form’  of  PBL  is  best  applied  to 
that  candidate,  measurement  issues  are  endemic.  After  all,  the  logistic  services  to  be 
outsourced  will  be  priced  contractually,  and  for  some  services,  there  is  no  clear  market 
to  determine  that  price.  When  discussing  CONUS  transportation,  prices  are  perhaps 
not  difficult  to  determine  by  reference  to  a  market.  However,  when  discussing 
something  like  intermediate-level  maintenance  of  a  deployed  weapon  system  on  which 
the  DoD  has  a  monopsony  and  the  number  of  qualified  bidders  is  quite  limited  (and  may 
indeed  be  only  one  or  two),  the  market  paradigm  clearly  breaks  down,  and  is  perhaps 
best  understood  in  the  context  of  game  theory  (Shubik  &  Levitan,  1980).  Unlike  the 
simple  solutions  of  monopolistic  games,  however,  the  monopsonistic  game  of  buying 
weapon  systems  logistics  is  hampered  by  the  difficulty  of  measuring  the  value  of  the 
services  to  be  obtained. 


In  discussing  whether  a  case  could  be  made  for  the  privatization  of  a  particular 
governmental  service,  Bendick  (1984)  said  it  was  important  to  compare  private  to 
‘nonmarket’  (i.e.,  organic)  alternatives,  and  that  the  private  sector  should  only  be 
employed  if  it  could  reasonably  be  expected  to  be  more  efficient.  He  listed 


“four  aspects  of  market  efficiency  [that]  are  important  to  examine: 

•  In  producing  the  services  ...  do  the  private  sector’s  production  processes 
and  input  costs  allow  it  to  generate  output  at  a  lower  total  cost  than  could 
the  public  sector? 

•  Are  the  administrative  costs  incurred  by  government  to  mobilize  and 
control  the  private  sector  les  than  the  cost  savings  from  more  efficient 
production? 
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•  Is  the  supply  side  of  the  market  sufficiently  responsive  that  private  firms 
enter  markets  rapidly  and  smoothly? 

•  Are  purchasers  sufficiently  rational  and  careful,  and  the  quality  of  the 
service  sufficiently  definable  and  measurable,  that  effective,  informed 
consumer  sovereignty  can  be  exercised?  (Bendick,  1984,  pp.  153-154).” 

Each  of  these  considerations  is  potentially  problematic  when  examining  PBL 
initiatives.  When  considering  the  first  of  the  above  factors,  the  existence  of  PBL 
contracts  in  which  the  private  sector  vendor  has  hired  back  organic  resources  as 
subcontractors  to  do  the  touch  labor  puts  in  question  exactly  what  services  are  being 
outsourced  -  logistics  or  management?  When  considering  the  third  of  his  factors,  the 
consolidation  of  the  defense  industry  and  the  decline  of  the  number  of  independent 
companies  that  might  act  as  potential  bidders  raise  concern.  However,  this  paper  will 
primarily  concern  itself  with  the  second,  and  especially  the  fourth  of  these  factors.  We 
will  discuss  how  an  excess  of  measurement  can  make  administration  of  comprehensive 
PBL  contracts  more  costly,  while  the  difficulty  with  defining  and  measuring  some  logistic 
services  make  consumer  sovereignty  difficult  to  establish. 

The  rest  of  the  paper  will  be  organized  as  follows.  First,  we  will  lay  out  a 
structural  framework  upon  which  measurement  issues  will  be  developed.  Upon  that 
framework,  we  will  then  develop  questions  about  how  measurement  informs  which  sorts 
of  candidates  are  best  suited  for  PBL.  Finally,  we  will  discuss  how  measurement  issues 
should  be  considered  when  deciding  on  the  form  of  PBL  to  be  adopted  for  a  particular 
candidate,  and  the  management  of  ongoing  PBL  contracts.  We  are  not  attempting  to 
clearly  delineate  between  good  and  bad  measures,  or  good  or  bad  candidates  for  PBL. 
Rather,  we  are  attempting  to  surface  imbedded  measurement-related  issues  that  may 
make  the  difference  between  a  problematic  implementation  and  an  easy  one.  Thus, 
this  paper  is  not  intended  as  a  guidebook  for  implementation,  but  rather  as  a  guidebook 
for  further  investigation. 
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A  Hierarchical  Bridge  Framework  of  Measurement 
for  PBL 


When  describing  logistics  service  acquisition  for  a  weapon  system  as  an 
economic  game,  it  is  important  at  the  outset  to  note  the  dissimilarity  between  the  two 
players.  The  vendor  has  a  clear  objective  of  maximizing  the  wealth  of  its  owners,  and  a 
clear  profit  incentive  (again,  we  assume  throughout  the  paper  that  we  are  dealing  with  a 
private-sector  vendor).  The  objective  of  the  user  acquiring  the  service  is  not  so  easy  to 
state,  and  far  more  difficult  to  assess.  Ambiguities  in  goals  and  a  lack  of  linkage 
between  services  acquisition  goals  and  strategic  objectives  are  intrinsic  aspects  of  the 
services  acquisition  process  (Camm  et  al,  2004;  Ausink  et  al  2002).  Maximizing 
national  security  would  be  one  way  to  state  the  objective,  and  the  incentive  (at  least  at 
the  organizational  level)  might  be  understood  in  the  same  terms  -  to  gain  more  security 
for  the  nation.  At  the  outset  then,  the  game  has  a  measurement  and  a  translation 
problem  -  measuring  the  services  in  terms  of  their  contribution  to  the  objectives  and 
incentives  of  the  DoD,  and  translating  that  measure  into  the  dollar  measurement  used 
by  the  private  sector. 


Of  course,  it  might  be  claimed  that  business  does  not  really  have  such  clear 
objectives  and  incentives  either.  There  is  a  venerable  literature  pointing  out  the 
maximization  of  shareholder  wealth  should  not  be  (and  is  not  in  practice)  the  sole  aim  of 
a  public  corporation.  Stakeholder  analysis  has  its  roots  in  this  observation  (Donaldson 
&  Preston,  1995).  But  even  stakeholder  analysis  (in  narrow  form  at  least)  does  not 
deny  the  centrality  of  profit  as  a  corporate  incentive,  rather  the  discussion  centers  on 
rights  of  resource  holders,  and  equitable  distribution  of  profits. 


The  management  fashion  of  Balanced  Scorecards  has  demonstrated  the 
willingness  of  corporate  executives  to  look  beyond  profit  in  analyzing  performance 
(Kaplan  &  Norton,  1992).  But  it  would  be  a  mistake  to  take  the  current  proliferation  of 
Balanced  Scorecards  as  evidence  that  corporations  suffer  under  the  same  sorts  of 
fundamental  measurement  problems  with  their  objectives  and  incentives  as  the  DoD. 
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The  Balanced  Scorecard  is  clearly  meant  to  be  a  diagnostic  tool  to  inform  management 
decisions  beyond  retrospective  financial  figures  about  the  long  term  viability  of  the  firm 
(i.e.,  it  is  meant  in  part  to  help  predict  and  control  future  financial  performance).  Kaplan 
and  Norton  (1992)  discuss  the  shortcomings  of  financial  performance  measures  in 
terms  of  their  ability  to  guide  (1 )  the  innovation  necessary  to  obtain  future  profitability, 

(2)  the  diagnosis  of  internal  process  problems  that  limit  current  and  future  profitability 
and  (3)  the  relationship  with  the  customer  necessary  to  sustain  future  profitability.  Their 
main  criticisms  of  current  financial  measures  (which  are  a  part  of  the  Balanced 
Scorecard)  are  that  they  are  historical  and  external  to  operations.  They  tell  a  firm  how 
well  it  has  performed,  not  why,  or  what  to  do  next  to  maintain  or  improve  future 
performance. 


But  measurement-related  differences  between  the  DoD  and  the  corporate  world 
exist  not  only  in  the  incentives  and  objectives  of  each,  but  also  in  the  process 
capabilities  that  are  important  in  developing  logistics  tactics  to  meet  those  objectives.  In 
reviewing  essential  dimensions  to  be  considered  in  logistic  performance  analysis  in  the 
commercial  sector,  Mentzer  &  Konrad  (1991)  developed  a  matrix  in  which  five  core 
logistics  functions  (transportation,  warehousing,  inventory,  order  processing  and 
administration)  could  be  measured  along  six  dimensions  (cost,  labor,  facilities, 
equipment,  time  &  energy).  Distinguish  between  those  dimensions  and  the  four 
“overarching  goals  of  PBL  ...  to  compress  the  supply  chain,  eliminate  non-value  added 
steps,  reduce  Total  Ownership  Cost  and  improve  readiness  for  weapons 
systems... (Department  of  Defense  -  Defense  Contract  Management  Agency  [DoD- 
DCMA],  undated)”  to  which  one  should  add  “increased  reliability  and  reduced  logistics 
footprint  (Office  of  the  Secretary  of  Defense  [OSD],  2003)”.  Aside  from  cost,  these  sets 
of  six  factors  seem  to  have  little  in  common.  But  all  of  the  commercial  sector  factors 
can  be  translated  into  cost,  and  can  be  understood  as  the  essential  dimensions  that 
must  be  managed  efficiently  and  effectively,  in  order  to  facilitate  logistics  support  of  the 
firm’s  profitability  objective.  The  DoD  factors,  on  the  other  hand,  do  not  all  translate  so 
readily  into  cost,  and  fall  into  three  categories  of  dimensions  that  logistics  improves 
warfighting  capability:  improved  readiness  (facilitated  both  directly  by  a  focus  on 
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readiness  and  indirectly  by  a  focus  on  reliability),  increased  agility  (reducing  logistical 
footprint,  eliminating  non-value-added  steps,  supply  chain  compression,  and  improved 
reliability)  and  reducing  cost  (by  freeing  capital  for  other  warfighting  priorities). 

This  is  a  significant  difference  in  how  logistics  is  viewed.  The  concept  of 
readiness  shows  up  as  ‘equipment’  to  commercial  firms,  who  view  the  maintenance  and 
functioning  (and  depreciation)  of  their  operating  capital  primarily  as  a  financial  question 
-  when  will  it  become  so  expensive  to  maintain  that  I  will  have  to  replace  it?  Since  DoD 
weapon  systems  are  often  quite  old,  very  expensive  and  difficult  to  re-capitalize  (lacking 
a  depreciation  mechanism,  recapitalization  is  often  driven  by  technological 
obsolescence  or  budget  constraints),  readiness  is  a  much  more  central  issue. 
Improvements  in  readiness,  of  course,  improve  warfighting  capability;  but  marginal 
improvements  are  quite  difficult  to  value  in  dollar  terms.  The  idea  of  ‘agility’  is 
increasingly  important  to  commercial  firms,  but  agility  in  a  commercial  operation  means, 
for  example,  the  flexibility  to  quickly  change  production  volumes,  or  quickly  changing 
production  technology.  It  shows  up  in  the  list  above  as  ‘time’,  which  is  also  translatable 
to  dollars.  DoD  operations  on  the  other  hand  are  mobile,  and  mobility  directly  impacts 
their  effectiveness.  Agility  is  not  a  newly  discovered  competitive  dimension  -  it  has 
always  been  an  operational  necessity.  Once  again,  however,  the  operational 
effectiveness  derived  from  a  marginal  improvement  in  logistics  agility  is  very  difficult  to 
translate  into  dollars. 


These  differences  in  organizational  objectives  and  the  consequent  logistics 
objectives  further  devolve  into  differences  in  process  measurement.  Caplice  &  Sheffi 
(1994),  in  a  classification  and  review  of  corporate  logistic  process  metrics  develop  three 
categories:  utilization,  productivity,  and  effectiveness  (see  Table  1).  Utilization 
measures  simply  address  the  question  of  how  much  of  a  resource  is  used,  compared  to 
what  has  been  made  available.  While  these  sorts  of  measures  may  be  useful  in 
assessing  the  efficiency  of  a  narrow  segment  of  a  process  (e.g.,  space  utilization  may 
be  useful  in  assessing  the  efficiency  of  a  facilities  layout  manager),  they  have  virtually 
no  contribution  to  the  understanding  of  logistics  contribution  to  organizational  objectives, 
primarily  because  they  do  not  measure  outputs  at  all.  It  might  be  claimed  that  they 
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measure  waste,  but  even  this  is  not  true  -  all  they  measure  is  activity,  not  whether  that 
activity  is  directed  toward  some  valued  outcome.  What  Caplice  and  Sheffi  (1994)  have 
called  effectiveness  measures,  on  the  other  hand,  ‘beg  the  question’  in  an  essential  way 
-  those  measures  are  only  as  good  as  the  norms  one  establishes  for  outputs.  They 
may  be  useful  for  historical  comparison  of  a  single  process,  but  their  value  in  comparing 
across  processes  or  in  guiding  resource  allocation  decisions  is  quite  limited. 

Productivity  measures,  on  the  other  hand,  incorporate  both  outputs  and  inputs.  For  the 
corporation,  assessing  the  contribution  of  an  activity  to  its  objectives  is  a  matter  of 
relating  those  inputs  outputs  to  profits.  While  of  course  this  is  not  necessarily  easy 
(e.g.,  single  factor  productivity  measures  do  not  capture  a  comprehensive  cost  picture), 
at  least  the  examples  given  by  Caplice  &  Sheffi  (1994)  can  be  measured  or  translated  to 
dollars  (e.g.,  dollars  paid  for  orders  processed,  or  shipments  made),  and  this  is  broadly 
true  of  metrics  proposed  in  other  reviews  of  corporate  logistic  performance 
measurement  systems  as  well  (e.g.,  Chow,  Heaver  &  Henriksson,  1994;  Lambert  & 
Burduroglu,  2000;  Mentzer  &  Konrad,  1991),  with  the  important  exception  of  customer 
satisfaction  metrics.  The  importance  of  the  ‘customer  view’  has  already  been 
mentioned  in  relation  to  Balanced  Scorecards,  is  often  mentioned  by  authors  on  logistic 
performance  measurement  (e.g.,  Mentzer,  Flint  and  Kent,  1999).  However,  it  is  worth 
noting  that  Lambert  &  Burduroglu  (2000)  list  “reliance  on  management  outside  of 
logistics  to  identify  the  impact  [of  customer  satisfaction]  on  revenues,  which  typically 
does  not  happen”  as  a  primary  disadvantage  of  customer  satisfaction  measurement. 
Hence,  beyond  simple  utilization  measures,  corporate  logistic  performance  measures 
can,  or  are  desired  to  be,  understood  in  terms  of  their  impact  on  profitability. 
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Table  1.  Corporate  Logistics  Metrics  (from  Caplice  and  Sheffi,  1994) 


Dimension 

Form  of  Metric 

Logistics  Examples 

Utilization 

Actual  Input  / 

Input  Norm 

Labor  hour  used  /  labor  hours  budgeted 

Area  of  warehouse  occupied  /  total  area 

Hours  machine  used  /  machine  capacity 

Productivity 

Actual  Output  / 
Actual  Input 

Ton-miles  delivered  /  costs  incurred 

Orders  processed  /  hours  of  labor 

Pallets  unloaded  /  hour  of  dock  time 

Effectiveness 

Actual  Output  / 

Output  Norm 

Items  filled  /  items  requested 

Shipments  on  time  /  shipments  sent 

Transactions  w/o  error  /  total  transactions 

Compare  those  corporate  logistic  measures  to  what  might  be  proposed  as  a 
productivity  ratio  for  weapons  systems  logistics  - 

Operational  Availability  (Ao)  of  deployed  systems  /  Total  deployed  hours. 


At  first  glance,  this  looks  like  a  utilization  measure,  not  a  productivity  measure  - 
but  Ao  is  often  used  as  a  surrogate  for  readiness,  which  is  typically  given  as  a  primary 
outcome  objective  of  military  logistics  (it  would  be  a  utilization  measure  if,  e.g.,  flight 
hours  were  in  the  numerator).  The  denominator  translates  to  dollars  in  a  budget 
(whether  or  not  they  could  be  translated  to  an  actual  cost  is  another  issue).  But  the 
numerator  is  not  and  should  not  be  translatable  to  dollars,  because  profit  is  not  the 
objective.  While  measurable,  it  is  difficult  to  value  in  terms  of  the  dollars  that  might  be 
spent  to  increase  it,  or  relinquished  in  order  to  pursue  other  priorities.  Another  problem 
is  that  Ao  is  only  a  surrogate  for  readiness  because  it  is  a  ‘single  factor’  measure.  It  is 
also  not  fine  grained  enough  for  many  resource  allocation  decision  we  wish  to  make 
(hence  the  distinction  between  ‘mission  capable’  and  ‘fully  mission  capable’  systems). 
And  finally  readiness  itself,  after  all,  is  only  a  surrogate  for  the  organizational  objectives 
of  the  DoD  (i.e.,  ready  for  what?).  Note  that  if  Ao  were  really  the  objective,  it  could  be 
maximized  by  parking  equipment.  Hence,  logistic  performance  of  weapon  systems  is 
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more  difficult  to  measure  than  commercial  logistics  (at  least  in  terms  of  productivity), 
and  perhaps  more  attenuated  from  DoD  objectives  than  are  commercial  logistic 
measurement  systems. 


Figure  1 .  A  hierarchical  framework  of  measurement  issues 


How  do  these  measurement  issues  inform  the  decision  to  bring  forward  a 
weapon  system  or  component  as  a  PBL  Candidate?  First,  again  considering  only 
outsourced  PBL  solutions,  we  must  consider  the  economic  logic  behind  outsourcing. 
One  basic  economic  justification  of  outsourcing  is  the  tradeoff  of  economies  of  scale 
with  a  reduction  in  transaction  costs.  If  the  outsourced  service  can  be  performed  by  an 
organization  that  offers  similar  services  to  a  number  of  other  customers,  that 
organization  gains  economies  of  scale,  and  should  be  able  to  offer  the  service  more 
cheaply  than  if  it  were  done  by  the  outsourcing  organization  in  house.  The  price  that  is 
usually  paid  for  such  outsourced  services  is  usually  in  terms  of  increased  transaction 
costs  to  negotiate  price  and  services,  and  monitor  performance  (Gustafson,  Aubert, 
Rivard  &  Patry,  1996).  When  economies  of  scale  are  difficult  to  obtain,  as  with  a  unique 
weapon  system  requirement,  some  of  the  underlying  justification  for  outsourcing 
disappears.  On  the  other  hand,  high  internal  transaction  costs,  due  to  e.g.,  high 
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reporting  requirements,  or  inefficient  internal  controls  make  outsourcing  relatively  more 
attractive.  If  high  internal  transaction  costs  are  part  of  the  justification  for  outsourcing  a 
PBL  contract,  then  it  is  important  that  the  system  or  component  being  outsourced  avoid 
some  of  those  transaction  costs.  When  measurement  of  logistic  outcomes  (readiness, 
agility  and  cost)  is  more  difficult,  it  will  mean  higher  transaction  costs,  because 
performance  monitoring  systems  will  have  to  be  more  elaborate,  and  fair  prices  will  be 
more  difficult  to  determine  and  negotiate. 

One  way  to  make  pricing  and  performance  monitoring  easier  is  by  reference  to  a 
market  for  similar  services.  Hence,  in  prescribing  a  methodology  for  the  analysis  of 
performance  based  contracts  for  contract  managers,  market  research  is  indicated  as  a 
required  step  (OSD-DAR,  2000).  For  comprehensive  weapon  system  logistical  support, 
or  for  weapon  system-unique  components,  there  will  likely  be  no  ready  market  for 
maintenance,  or  many  other  logistical  support  functions.  In  those  cases,  the 
implementation  of  an  outsourced  PBL  solution  will  require  more  cost  and  effort  to 
develop  appropriate  metrics,  and  negotiate  appropriate  prices. 

In  summary,  measurement  issues  are  endemic  to  the  relationship  between 
commercial  sector  vendors  and  the  DoD.  From  the  point  of  view  of  measurement,  the 
best  PBL  candidates  are  those  with  external  markets  for  services,  and  clear  outcomes 
that  are  easy  to  relate  to  mission  objectives.  This  is  not  to  place  a  definite  boundary  on 
the  systems  where  PBL  ought  to  be  applied,  but  only  to  point  out  that  measurement 
issues  may  make  some  PBL  implementations  far  more  difficult  and  expensive,  and  may 
affect  the  form  of  the  PBL  solution. 
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Measurement,  the  PBL  support  spectrum  and  the 
management  of  ongoing  contracts 


One  of  the  characteristics  of  PBL  is  that  general  characterizations  are  hard  to 
make.  The  top  level  guidance  for  the  initiative  always  has  caveats  such  as 

“there  is  no  one-size-fits-all  approach  to  PBL.  Several  programs  have  started  the 
move  to  PBL  under  initiatives  designed  to  meet  the  programs’  specific 
requirements.  Each  program  has  tailored  the  PBL  application  to  its  unique 
circumstances  taking  into  account  cost,  schedule,  or  product  integrity  to  meet 
warfighter  capability.  (DoD,  2001,  p.  2-2)” 

In  reviewing  implementations,  a  wide  variety  of  approaches  can  be  found,  in 
terms  of  measurement  and  incentives,  and  in  terms  of  the  level  at  which  the  PBL 
contract  is  written:  from  a  complete  weapon  system  as  with  the  DDG-51 ,  to  component 
level  stock  support,  as  with  the  AEGIS.  The  spectrum  of  choices  is  usually  described  in 
terms  of  the  degree  of  commercial  support  involved,  and  a  frequently  encountered 
graphic  (which  we  have  been  unable  to  track  to  its  original  source)  is  shown  in  Figure  2. 


Figure  2.  The  PBL  Support  Spectrum. 


While  examples  of  systems  are  often  given  in  association  with  this  chart,  and 
definitions  of  the  various  anchor  points  (e.g.,  Mini-Stock  Point)  are  offered,  very  little 
guidance  can  be  found  for  the  program  manager  as  to  what  characteristics  of  a  weapon 
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system  should  inform  the  choice  of  the  degree  of  commercial  involvement,  and  whether 
the  contract  should  be  offered  at  the  system,  or  component  level.  It  is  our  position  that 
measurement  issues  should  inform  this  choice. 

A  primary  aspect  of  measurement  informing  the  choice  of  commercial 
involvement,  which  we  have  not  discussed  yet,  is  risk.  As  should  be  clear  by  reference 
to  Figure  1,  vendors  will  be  primarily  interested  in  reductions  of  financial  risk,  while  the 
DoD  is  entirely  concerned  with  operational  risk.  The  tradeoff  of  these  two  kinds  of  risk 
is  central  to  the  logic  of  PBL  outsourcing.  Contracts  are  almost  always  offered  across 
multiple  years  (lowering  financial  risk  for  the  vendor),  with  the  expectation  that  the 
vendor  will  assume  some  degree  of  operational  risk.  Figure  3  shows  the  expected 
tradeoff  of  operational  risk. 

Figure  3.  Intended  risk  transfer  under  PBL.  From  DoD-DCMA  (undated). 
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Although  risk  is  clearly  indicated  as  a  factor  to  consider  when  developing  a  PBL 
strategy  (ASN-RDA,  2003;  Office  of  the  Assistant  Deputy  Under  Secretary  of  Defense  - 
Logistics  &  Material  Readiness  [OSD-LMR],  Undated),  this  factor  is  rarely  mentioned  as 
a  candidate  for  measures  of  ongoing  performance  in  PBL  contracts.  Indeed,  it  has 
been  said  that 


“minimal  contract  management  involvement  is  anticipated  as  long  as  the 
contractor  meets  contractually  specified  performance  metrics.  However  our 
involvement  may  increase  if  the  contractor  systems  and  processes  are  not 
functioning  correctly  and  end  users  are  not  appropriately  supported  (DoD-DCMA, 
undated,  pp  28-29.)” 

This  is  a  curious  form  of  risk  transfer.  Operational  risk,  as  we  understand  it, 
involves  the  variance  associated  with  outcomes.  The  assumption  of  operational  risk  by 
a  contractor  then  would  entail  accountability  for  control  of  that  variance,  and  assurance 
(with  appropriate  remedy  in  the  case  of  failure)  of  the  mitigation  of  its  impact.  Here, 
rather,  it  seems  that  vendors  are  being  asked  to  assume  some  aspects  of  risk  (not 
clearly  defined),  but  that  the  DoD  will  ‘increase  involvement  if  the  contractor’  fails  to 
meet  requirements.  We  do  not  claim  that  this  is  risk  transfer  in  name  only,  but  that  the 
form  of  risk  being  assumed  by  vendors  is  not  what  is  commonly  understood  as 
operational  risk,  and  the  degree  of  risk  they  are  assuming  is  apparently  quite  limited. 


It  is  our  view  that  the  degree  of  operational  risk  a  contractor  can  assume  is 
limited  in  many  cases  by  the  nature  of  military  operations.  It  is  unrealistic,  for  example, 
to  assume  that  contractors  will  be  able  to  perform  operational  level  maintenance  on  a 
ground  combat  weapon  system;  difficult  issues  relating  to  the  physical  risk,  insurance, 
and  liability  of  non-DoD  personnel  in  or  near  combat  need  to  be  addressed.  These 
sorts  of  operational  risk  are  difficult  to  measure,  and  even  more  difficult  to  value.  We 
think  it  likely  therefore,  that  commercial  sector  vendors  will  be  reluctant  to  undertake 
these  risks.  Depot  level  maintenance,  and  operation  of  CONUS  inventory  control  points 
involve  less  operational  risk,  and  risk  in  a  form  that  is  easier  to  measure,  and  less  costly 
for  a  vendor  to  assume,  and  hence  we  think  it  likely  that  the  more  operational  risk 
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involved  in  the  logistical  support  of  a  particular  system,  the  more  organic  resources  will 
need  to  be  involved. 


Proposition  1.  When  operational  risk  is  high  or  difficult  to  measure,  PBL 
strategies  should  seek  less  commercial  sector  involvement. 

Within  the  context  of  a  price  negotiation,  it  is  also  key  to  understand  the  benefit 
the  DoD  provides  by  eliminating  financial  risk,  as  this  is  part  of  what  we  are  paying  to 
potential  vendors.  Especially  if  interest  rates  and  rise  and  the  difference  between  the 
cost  of  capital  and  risk-free  rates  increase,  what  the  DoD  offers  in  terms  of  financial  risk 
mitigation  is  highly  valuable.  This  valuable  benefit  is  not  free  for  the  government  to 
offer,  and  should  be  incorporated  into  pricing  and  contract  negotiations.  If  less 
operational  risk  is  assumed  by  the  vendor  (or  if  that  risk  is  difficult  to  assess),  less 
financial  risk  should  be  mitigated  -  meaning  contract  terms  should  be  reduced. 


Proposition  2.  When  commercial  sector  vendors  assume  less  (measurable) 
operational  risk  under  a  PBL  contract,  the  term  of  that  contract  should  be  less. 

On  the  other  hand,  the  outcomes  of  PBL  strategies  involving  only  certain 
components,  or  only  depot-level  support,  are  more  difficult  to  tie  to  weapon  system 
outcomes.  Consider  Figure  4,  which  shows  a  highly  stylized  and  simplified  version  of  a 
weapon  system  and  its  major  components,  along  with  the  failure  rates  (mean  time  to 
failure)  of  each  of  the  components.  Assuming  failure  of  any  of  the  components  cause 
the  weapon  system  to  become  non-mission  capable,  the  failure  rate  of  the  overall 
weapon  system  is  then  an  order  statistic,  formed  of  the  distributions  of  the  time  to  failure 
of  all  of  the  components.  Now  consider  the  problem  faced  by  a  program  manager  who 
has  decided  that  his  best  PBL  strategy  involves  outsourcing  only  component  A  (the  one 
with  the  highest  failure  rate).  To  properly  value  the  impact  of,  for  example  a  proposed 
incentive  to  improve  the  reliability  of  component  A  by  10%,  the  program  manager  would 
need  not  only  distributional  information  about  the  time  to  fail  of  all  the  other 
components,  but  also  a  working  model  which  imbeds  that  system  in  mission 
requirements.  After  all,  the  final  value  of  an  improvement  in  reliability  of  a  component 
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(to  readiness  -  of  course  there  are  other  benefits  in  terms  of  reduced  life-cycle  cost  of 
spares,  and  improved  agility  through  reduced  footprint)  rests  in  the  increased  likelihood 
of  mission  success  in  the  deployed  weapon  system.  The  sort  of  integrated  simulation 
model  needed  to  properly  assess  the  impact  of  improved  component  reliability  would  be 
expensive,  and  more  importantly,  time  consuming  to  build.  Given  the  time  pressures 
put  upon  Program  Managers,  it  is  easy  to  see  that  the  situation  is  problematic. 


Proposition  3.  PBL  strategies  involving  less  than  comprehensive  logistical 
support  of  a  weapon  system  (e.g.,  for  a  component)  should  nonetheless  have 
integrated  weapons  system  models  in  support  of  their  business  case  analysis. 

In  summary,  measurement  issues  exist  across  the  PBL  spectrum,  but  present 
different  sorts  of  challenges  at  either  end.  Ultimately  there  are  at  least  two  core 
measurement  issues  that  should  be  referred  to  when  deciding  on  an  appropriate  level  of 
support  within  the  PBL  spectrum.  The  first  is  the  valuation  of  outcome-related 
performance,  and  the  second  is  valuation  of  operational  and  financial  risk.  While 
outcomes  are  easier  to  measure  at  the  right  end  of  the  spectrum,  one  is  less  likely  to 
find  a  relevant  market  to  support  price  and  value  decisions.  On  the  left  hand  of  the 
spectrum,  markets  may  well  exist  that  essentially  duplicate,  for  example,  the  services  of 
a  mini-stock  point.  However,  the  valuation  of  those  isolated  services  in  terms  of 
weapon  systems  performance  is  even  more  difficult. 

Figure  4.  Need  for  integrated  system  model  to  judge  impact  of  component  outcomes. 
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Aside  from  risk,  another  distinction  important  to  develop  when  discussing 
management  of  ongoing  PBL  contracts  is  the  difference  between  process  and  outcome 
measures.  It  is  our  position  that,  while  PBL  is  clearly  intended  to  buy  outcomes,  and 
relieve  management  of  the  necessity  of  monitoring  the  details  of  ‘how’  performance  is 
obtained,  a  great  deal  of  effort  is  still  being  devoted  to  process  measurement.  It  has 
recently  been  said  that  ‘too  many  metrics’  is  a  major  problem  with  PBL  implementation 
(Office  of  the  Secretary  of  Defense,  Undersecretary  for  Acquisition,  Technology  & 
Logistics  [OSD-ATL],  2004).  If  the  DoD  is  buying  outcomes,  not  process,  then  it  may  be 
that  much  of  the  process  measurement  is  unnecessary. 

An  example  of  ‘too  many  metrics’  is  shown  in  Figure  5.  This  is  a  slide  shown  in  a 
brief  to  a  base  commander  to  provide  an  overview  of  the  PBL  contracts  at  his  base. 

This  is  a  small  base,  with  only  a  handful  of  PBL  contracts.  Clearly,  the  commander 
understands  that  there  are  too  many  metrics,  and  is  tracking  them  quarterly  in  order  to 
push  for  their  reduction.  Here,  the  number  of  metrics  itself  has  become  a  metric,  with 
visibility  to  the  top  operating  officer  at  a  command. 


Figure  5.  A  measure  of  PBL  measures  used  at  one  DoD  command. 
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Exactly  how  the  superabundance  of  metrics  arises  is  an  open  question.  PBL  is  a 
process  meant  to  streamline  management  concern  with  the  details  of  a  logistical 
process.  In  part,  it  may  arise  from  a  broader  context  within  DoD,  of  understanding  the 
systemic  relationships  of  which  logistics  is  only  a  part.  Under  various  titles,  including 
Integrated  Logistics  Support,  the  last  several  years  has  seen  an  increasing  awareness 
of  the  embedded  nature  of  logistical  support,  and  the  inter-relationships  involved 
between  e.g.,  manpower,  maintenance,  and  supply.  Figure  6  is  taken  from  a 
presentation  to  a  PBL  ‘tiger  team’  concerned  with  establishing  metrics.  Figure  6a  lists 
the  ‘balanced  scorecard’  of  top  level  factors  for  weapon  system  support. 

Figure  6a.  DoD  Balanced  Scorecard  for  Weapon  System  Support.  From  PRTM  (2004). 
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Figure  6b,  translates  the  scorecard  into  logistics  measures,  by  mapping  between 
the  high  level  scorecard  factors,  and  the  primary  factors  (process  elements)  of  another 
management  fashion  called  Integrated  Logistical  Support  (ILS).  The  details  of  ILS 
process  elements  are  not  germane  here,  only  to  note  that  they  are  indeed  detailed  and 
process  oriented.  If  one  starts  with  a  multidimensional  balanced  scorecard,  and  works 
through  these  process  elements,  it  is  easy  to  see  how  the  number  of  metrics  that  must 
be  tracked  could  be  numbered  in  the  hundreds  -  in  fact,  it  would  be  hard  to  avoid.  One 
must  ask  why  however,  if  we  are  engaged  in  an  initiative  to  buy  performance,  we  are 
starting  with  a  detailed  map  of  the  internals  of  the  process.  Wouldn’t  it  make  more 
sense  to  measure  only  key  outcomes,  and  measure  them  well? 


When  we  first  presented  this  idea  at  a  conference,  we  were  met  with  the 
objection  that  an  abundance  of  measures  do  not  necessarily  distract  a  decision  maker 
from  key  tasks.  The  analogy  was  drawn  to  a  pilot  in  a  jet,  where  the  cockpit  has  a 
superabundance  of  meters  and  instruments,  almost  all  of  which  can  be  ignored,  except 
in  the  case  of  an  emergency.  The  analogy  is  a  telling  one,  in  that  most  of  the  people 
making  decisions  about  metrics  for  PBL  have  themselves  been  pilots,  or  ship  captains, 
or  in  charge  of  some  complex  process  in  the  past.  However,  PBL  is  not  supposed  to 
present  the  DoD  with  a  complex  process  to  manage  -  it  is  supposed  to  take  one  off  the 
hands  of  the  DoD.  We  aren’t  supposed  to  be  flying  the  plane  -  we  are  supposed  to  be 
passengers.  When  you  are  paying  someone  else  to  get  you  to  your  destination,  you 
care  about  the  price  of  the  ticket,  and  arriving  on  time. 


Of  course  this  is  a  simplification.  When  discussing  a  PBL  contract  with  one 
deployed  squadron,  we  were  met  with  a  complaint  about  the  difficulty  of  obtaining 
requisition  status  for  parts  that  were  backordered.  The  maintenance  officer  for  that 
squadron  was  not  impressed  that  the  average  lead  time  to  get  the  parts  had  been 
reduced,  because  his  primary  decision  when  confronted  with  a  backorder  was  whether 
or  not  to  devote  the  manpower  to  strip  a  spare  part  from  another  downed  aircraft.  It  was 
a  decision  that  was  difficult  to  make,  without  knowing  how  long  it  would  be  before  the 
spare  part  arrived.  Clearly,  in  this  case,  some  visibility  into  the  process  was  needed, 
but  only  because  a  key  operational  decision  rested  on  the  measurement  of  that 
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process.  Given  that  at  least  part  of  the  justification  for  PBL  is  meant  to  be  a  reduction  in 
transaction  costs,  we  think  this  should  probably  be  a  general  rule. 

Proposition  4.  In  the  management  of  ongoing  PBL  contracts,  metrics  should 
primarily  concern  themselves  with  valued  outcomes,  and  should  be  related  to 
weapon  system  cost,  readiness,  and  agility.  Process  measures  should  only  be 
applied  when  key  operating  decisions  depend  on  the  status  of  the  process  itself. 

Figure  6b.  Metric  Areas  informed  by  Balanced  Scorecard,  and  ILS  Process  Elements. 
From  PRTM  (2004). 
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Finally,  we  turn  to  the  measurement  of  risk  in  ongoing  PBL  contracts.  Operational 
risk  is  always  difficult  to  assess.  In  the  context  of  support  for  a  weapon  system,  it  can 
be  understood  as  variance  in  the  logistics-related  readiness  of  that  system.  A  common 
measure  of  readiness  is  Ao.  To  see  how  variability,  or  risk,  affects  Ao,  consider  Table 
2.  The  table  shows  the  availability  of  two  squadrons  of  10  aircraft  over  a  20  day  period. 
Over  that  period,  both  squadrons  would  report  operational  availability  of  95%.  But 
consider  that  a  mission  requiring  10  aircraft  would  be  degraded  (more  likely  to  fail)  only 
5%  of  the  time  in  the  first  case,  but  50%  of  the  time  in  the  second  case.  The  difference 
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is  the  variance  in  Ao.  To  our  knowledge,  in  spite  of  the  exhortations  of  the  centrality  of 
risk  assumption  and  readiness  to  PBL,  there  are  no  programs  currently  tracking  this 
measure. 


Table  2.  Distributional  differences  in  a  95%  Ao. 
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In  a  very  thoughtful  document,  the  office  of  the  Assistant  Secretary  of  the  Navy 
for  Research,  Development  and  Acquisition  (ASN-RDA,  2003)  listed  factors  to  consider 
when  deciding  whether  to  use  a  PBL  strategy:  life-cycle  stage,  acquisition  program 
strategy  (including  programmatic  risks),  organic  impact  (e.g.,  maintaining  engineering 
expertise),  commercial  base  (including  additional  risk  required  of  industry  partners), 
design  considerations  (including  risk  associated  with  incentives  and  performance 
thresholds),  and  technology  considerations  (including  supportability  risks).  Although 
risk  is  mentioned  in  four  of  the  six  factors,  there  is  no  mention  of  the  sort  of  operational 
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risk  discussed  above,  and  shown  in  Figure  3.  Indeed,  there  is  little  mention  of  risk  for 
ongoing  contracts  in  this,  or  any  other  guidance  documents  for  PBL.  We  find  this 
curious.  If  part  of  what  we  are  buying  is  operational  risk  mitigation  (in  key  performance 
dimensions),  it  seems  to  us  that  it  ought  to  be  measured. 

Propositions  5.  Operational  risk  (variability)  in  key  performance  measures  should 
be  tracked  for  ongoing  PBL  contracts.  Where  essential  to  mission  support,  a 
reduction  in  variability  should  be  supported  with  appropriate  incentives. 
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Conclusion 


Organic  acquisition  and  sustaining  logistics  in  the  Department  of  Defense  of 
today  is  seriously  challenged  in  its  ability  to  provide  significantly  higher  levels  of 
operational  availability  and  effectiveness  commensurate  with  the  complexity  and  high 
costs  of  weapon  systems.  In  addition  we  exacerbate  the  pressure  on  logistics  support 
as  we  prosecute  our  national  security  strategies  with  ships,  airplanes,  and  equipment 
that  are  aging  when  a  sufficient  replenishment  rate  to  counter  that  aging  is  unaffordable. 
Any  change  in  the  Defense  Department  Logistics  discipline  that  can  produce  greater 
utility  at  lower  costs  should  be  the  focus  of  an  energetic  pursuit.  In  order  to  bring  about 
the  necessary  improvements  we  must  mitigate  cultural  impediments  and  concurrently 
align  measurement  and  rewards  with  good  logistics  performance  in  the  acquisition 
process  and  operation  of  weapons  systems  in  the  field.  It  is  also  important  to  fully  fund 
logistics  requirements,  safeguard  the  funding  and  engage  in  long-term  contracts. 

The  Department  of  Defense  faces  major  challenges  in  establishing  a  coherent 
logistics  support  strategy  for  its  weapon  systems.  While  Performance-Based  Logistics 
has  been  decreed  as  a  preferred  implementation  strategy,  real  questions  remain 
unanswered  about  objectives  and  measurability.  The  disaggregation  of  logistics 
support,  and  the  emphasis  on  an  increased  role  for  the  private  sector  in  logistics,  also 
raise  major  issues  related  to  the  assumption  of  risk.  In  making  sourcing  decisions,  DoD 
must  act  as  a  “knowledgeable  client”  and  avoid  simplistic  decisions  that  may  lead  to 
critical  systems  being  left  unsupported. 

In  summary,  this  paper  has  presented  a  framework,  and  propositions  about  the 
impact  of  measurement  on  PBL.  None  of  the  propositions  have  the  status  of  fully 
supported  hypotheses,  or  fully  developed  theorems.  All  need  further  investigation. 

Some  of  the  propositions  are  empirical,  and  need  to  be  investigated  in  the  field.  Others 
are  prescriptive,  and  need  to  be  supported  by  modeling  and  analysis.  Our  hope  is  that 
we  have  furthered  the  discussion  of  metrics  for  PBL,  and  added  to  the  momentum  for 
improved  implementations  of  PBL. 
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